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ABSTRACT 

The use of narrative test reports overcomes the major 
carrier to understanding reports, understanding the language that is 
used. Early attempts to utilize the computer in generating narrative 
reports include: (1) Teaching Information Processing System (TIPS), 
involving periodic collection of information from students regarding 
courses, which is summarized within a few hours into three types of 
reports — student, section learer, and professor; (2) Preliminary 
Scholastic Aptitude Test (PSAT) Score Reports, involving eighty 
distinct sentences, in which variable phrases might be embedded, 
which are used to compose 75 distinct paragraphs, which in turn are 
combined to produce the 100 letters needed to interpret all 
combinations of scores; (3) Programmed Composition of Psychological 
Test Reports, involving selection of one of eight possible statements 
for each of the 101 scales of the MMPI (or the 124 of the MMPI and 
the CPI) • Arguments can be made for and against the use of narrative 
reports. (KM) 
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When a person takes a test (or in some way is formally 
assessed), a report is usually produced. The report may be simply 
the number of correct items on the test, or it may be a detailed 
description of the testee's responses to a psychological test 
battery v.'ith comments and interpretation from a psychologist. The 
purpose of assessment reports is commurvicaAloiX^^^f ^nf ormation for 
decision-makinq , and the communication must be understandable to 
the receiver of the report* 

Assuming no difficulties with the validity and reliability of 
the test, and a general understanding of the purpose and method of 
the test, the major barrier to understanding a performance report 
is understanding the language that is used. The language is 
usually numerical values of quantitative concepts such as rav/ 
scores, grade-equivalent scores, stanines, percentiles and standard 
errors of measurement. Certainly the target audience of the 
reports (students or parents or instructors or counselors, etc. ) 
determines the kind of concern that language problems elicit; hvt 
to err by assuming an undistorted transmission of information is 
easy and a common occurrence. 

The result of an understandable report should be the transfer 
of information on performance that can be used in the decision- 
making process. In the case of standardized achievement tests in 
education, Goslin found (1967, p. 32) that elementary teachers 
receive score reports on pupil performance about 80 per cent of llie 
time and have free access to these scores in virtually every case. 



yet. this information is shared routinely with pupils and their 
parents less than eight per cent of the time. Current test re- 
porting practices, then, seem to indicate that at least some of 
the information that could be useful in making educational decisions 
about a- child is not routinely shared with the child and his 
parents. Proceeding on the assumption that testing information 
should be shared with parents and children, the obvious prohi- 
biting factor is that~nhey~~w6 u I dnrnrtTTrnirersTancI the report." 
That is probably true. After all, some of our teachers and adminis- 
trators have difficulty understanding current test reports. An 
alternative to depriving parents and students of testing information 
is to produce a testing report that is understandable to them. The 
suggestion and theme of this symposium is the use of reports that 
are in a narrative format--words that blend to forn sentences 
v/hich join into paragraphs. 

tARLY PROJECTS 

Let's look at a few early attempts to utilize the computer in 
generating narrative reports. 
Teac hin g Informa t ion P r q ces sing System (TIPS) 

Kelley (1968), a professor of economics at the University of 
I'isconsin, developed a program, which he calls TIPS, to assist 
him in teaching. TIPS involves periodic collection of information 
from students regarding either their understanding of course 
materials or their reaction to various aspects of course presenta: 
tions. TIPS provides a means of efficiently utilizing this infor- 
nation for instructional purposes. The information, which is 



collected on specialized forms suitable for machine processing, 
is composed of student responses to a series of multiple-choice 
questions. Surveys of six to tv/elve questions take about five to 
ten minutes to administer. Within a few hours this information 
is processed and summarized in three separate reports: one for 
each student, one for each section leader, and a third for the 
professor. 

Tite--^ti«J^t-^rept^ of his performance: his 

response to each question, the correct answers, and the total 
number of his correct answers. On the basis of this information, 
assignments for the forthcoming period are also indicated. The 
assignments (some required--some optional) vary considerably in 
nature, level, and intensity. A student scoring well may receive 
optional assignments and/or required work at a higher level. 
The student performing poorly may receive not only a heavy dose 
of required v/ork but also a set of materials designed to bring 
him tov/ard the mean class performance. 

Additional informaticr. oii the student report is generated on 
the basis of past as well as current performance. If the student 
has performed poorly over several surveys, he will be instructed 
on the student report to establish an appointment with the instruc 
tor or teaching assistant. If the student performed consistently 
and exceptionally well, he may be notified that a short paper 
may be substi tuted» at his option*, -for the midterm examination. 
A sample of a student report is presented as Figure 1. 



Figure 1 about here 
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The teaching assistant report contains information to help him 
appraise the performance of his individual sections, including 
statistics on percentage correct by question or by concept, 
actual responses on the survey, lists of students required to 
establish appointments or tutorials, and so forth. 

The professor's report is similar to that received by the 
teaching assistant, although the Information a va fT£rb1^"arpp- i i ti !> i v — 
all students enrolled in the course rather than only to the 
students in particular sections. With this information the 
professor may elect to alter lectures, section coverage, problem 
sets, or other teaching Instruments for the forthcoming period. 
In summary, TIPS is a systam for gathering and reporting objective 
and timely information useful for more effective teaching. 

P-T eliminar y Scbol as t i c Aptit ude Test (PSAT) Score Reports 

The effective reporting of test results to admissions officers, 
guidance counselors, and the i i-d i vidua Is who take educational test*? 
is a rather more complex r.otter than might appear at first alance. 
It is apparent that reporting only numerical scores is hardly 
adequate. The statistical, psychological, and educational con- 
texts which allov/ the user to infer relevant meanings must be 
provided as well. The problem Is particularly acute for the 
programs in which tnt vimary reporting target is the test taker 
himsel f . 

Since the precise interpretation of mental test scores doss 
require rather sophisticated insights Into statistics, psychology, 
and education. It Is common practice to have test scores reported 
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to individual teat takers by guidance counselors who are expected 
to have the necessary sophistication. Unfortunately, not all 
guidance counselors do have sufficient psychometric sophistication, 
nor do they have the time to prepare detailed analyses and to 
give individual interpretation of mental test results. Of course, 
most educational testing services prepare a wide range of inter- 
pretive materials for their_J:e^sting programs to aid both the 
counselor and the student in interpreting test scores. But even 
with these aids, some statistical sophistication is still needed 
for adequate understanding. 

Helm and Harasymiw (1968) designed variable format computer- 
generated letters to be sent to examinees of the Preliminary 
Scholastic Aptitude Test as a report of their performance on the 
test. They found it necessary to prepare eighty distinct sen- 
tences where a sentence might have variable phrases imbedded. 
These eighty sentences were used to compose seventy-five distinc:: 
paragraphs which, in turn, w&»'e combined to produce the 100 
letters needed to interp-. ct all combinations of verbal and mathe- 
matical scores. They wrote a computer program to generate these 
letters and a sample of their report appears as Figure 2. 



Figure 2 about here 



Programmed Xpjmj.OALtJoA-P.f-.P§l g^o^o ^ ^c^^ ^^^^ Report s 

A computer-generated verbal diagnostic report on a standard i;:ed 
psychological test has been used routinely at the Mayo Clinic <n 



Rochester, Minnesota since 1962 (Sv/enson, 1962), and is available 
commercially from Behaviordyne, Inc. The Mayo program utilizes 
the Minnesota Multi-phasic Personality Inventory (MMPI) which is 
an objective pencil-paper psychological test. The machine- 
produced report is a group of disconnected statements, or decisions, 
about the subject as measured by the scales of the MMPI. Finney 
improj^d_iyi^JJ1a^^ an alternative of adding 

scales from the California Psychological Inventory (CPI), and 
improving the coherence of the report (1966). A large number of 
scales are scored, 101 with the MMPI alone or 124 with the MMPI 
and the CPI. 

The report is built by selecting statements and then combining 
them into paragraphs. For each of the 101 scales, one statement is 
chosen from among eight possible statements, depending on the 
individual's score on the scale. By this method, 101 statements 
are chosen from a repertory of 308. Finney's program has the 
computer comfose a full report on each individual's personality-- 
the "kind of report that a p'^^ychologist might write after seeing a 
person several times and administering a full battery of tests. 
The first one-third of a sample report is given as Figure 3. 



Figure 3 about here 



Finney and Auvenshire have subsequently developed several 
different kinds of reports written for different purposes. They 
are now extending their work to other objective psychological 
tests . 



Baker suggested (1971) that electronic computer technology 
be utilized" to generate reports on other types of testing instru- 
ments in order to make the results more meaningful to the persons 
examined and to facilitate better use of the results of testing. 
He concluded: 

The mechanics of having the computer program prepare 
verbal descriptions depends upon several factors. 

Fj-r-st,.the insight of the test constructor into the 

area of interest; second, the relation of levels of 
test and diagnostic scores to pupil performance; third, 
the cleverness of the computer programmer in generat- 
ing connecting prose from somewhat disconnected verbal 
descri ptions . 

Some Potential Advantages 

Depending upon the specific application, several advantages 
accrue from the use of narrative reports. A few general advan- 
tages follow: 

♦Clear Communication--The report is in understandable English and 

does not ask the recipient to search the page for clues on where 

to begin "reading" the repu.c. 
*Personal--The use of the testee's name and the appropriate personal 

pronouns can easily be included in the text to humanize the report 

and increase the attention paid to it. It does not "look like 

a computer report." 
*Ef f icient--A more complete report is available at the cost of fewer 

professional -person-hours. Narrative reports are not suggested 

as a substitute for the professional, but rather as an aide to 

the professional . 
*Self-!-xplanatory--The report can be reasonably self-contained 

with less need for individual guidance to assist in its initial 

interpretation. 



8 

*Flexible--The underlying philosophy of the narrative can be varied 
as needed for different audiences and audience sub-sets. 

^Public Relations--An inherent public relations value can accrue 
to the organization from a professional report that is under- 
standable and personalized. 

Certainly there are opposing arguments that could be offered: 

*Muddled Communi cati on--The report has to be read, thereby 
restricting its audience. Realizing that, the reading level of 
the report still makes communication less than efficient with 
another segment of people. Others who can read well, do not v^fant 
to be bothered to read--they just v/ant the score, and without 
extraneous verbiage. 

*Too Personal --Some people may react negatively to the thought 

* of a machine attempting to be "intimate" with them. 

-Expensi ve--Initial development of the system is expensive with 
no promise of a lowered cost per report. 

This list could also be extended, and perhaps you will add 
to it during the discussion after the papers. We do not pretend 
to propose a panacea for reporting problems. He do, however, 
think that narrative reports have a place, and that their appli- 
cation should be furthur explored to determine its precise location. 



Figure 1 

SAMPIE COPY OP KELLEY'S TEACHING INFORMATION 
IROCESSING SYSTEM (TIPS) STUDENT REPORT 

TIPS 

STIBENT PERFORMANCE SURVEY, RESULTS 
PRINCIPLES OF ECONOMICS (jD3) 
PROFESSOR ALLEN C. KELLEY 

/ LAWRENCE 



SIRVEY TAKEN JO/25/57 „ ^ ^ 
SECTION NtMBER AND TIME 2. 9:55 F 
SECTION LEADER MISS GREEN 

our OF A TOTAL OF 3 D QUESTIONS, Y OU CORRECTLY ANSWERED 3. THE QUESTION 
Nll«ER, YOUR RESPONSE AND THE CORRECT ANSWER ARE PROVIDED IN THE TABLE 
B ELOW. YOU ARE URGED TO MAKE SURE YOU UNDERSTAND THE NATURE OF ANY 
INCORRECT RESPONSES YOU MAY HAVE ^W£. 

SUWARY OF SURVEY RESULTS 
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SStS^LWHR^P ™^ WE^' BE HANDED IN DURING THE DISCUSSION 
SECTION ON 11/03/57, IS THE FOLLOWING — 

PROBLEMS L 3 AND 4 ON HANDOUT 2C 

ADDITIONALLY, YOU ARE REQUIRED TO WORK THROUGH CHAPTER 2 OF - MICRO- 
ECONOMICS, A PROGRAMf€D BOOK, BY LIMSEEN, AHIYEH, AND BACH. IT WOULD 
BE USEFU. TO CONSULT THE PROGRAMMED UNIT BEFORE VOU READ HANDOUT 2C. 

THE MATERIALS, YOU MAY, AT YOUR OPTION, 
ELECT TD COMPlfTE THE FOUXWING PROBLEMS — 



WORKBOOK,.PP. 37-38. , 
HANDOUT 2A, PROBLEMS 3 AND 4. 



^l^Z iS iSi BY PROFESSOR MILTON FRIEEMAN, PAST 

SSIFSI.SEJ"^-/*'^^^ ECONOMIC ASSOCIATION, WILL BE HELD IN 6m 
SP Jk «J2S!;;.i? P*"* * 2/0^^7. the topic - monetary and FISCAL 
POLICY RECONSIDERED. 
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SAMPIE COPY OF THE PRELIMINARY SCHOIASTIC 
APTITUDE TEST (PSAT) SCORE REPORT 

EDUCATIONAL TESTING SERVICE 
PRINCETON/ N.J. 085^0 

JANUARY 21/ H)6 

EEAR M?. L£TR 99/ 

WE WANT TO REPORT TO YOU THE SCORES YOU EARNED ON THE 
PffiUMINARY SCHOLASTIC APTITUDE TEST-YOU TOOK ON OCTOBER 9/ 
1965, YOUR APTITUDE FOR COLLEGE WORK IS OUTSTANDING. IF YOUR 
HIGH SCHOOL MARKS ARE CONSISTENT WITH THE HIGH SCORES YOU HAVE 
EARNED ON THE TEST YOU WILL HAVE LITTLE DIFFICULTY IN BEING 
ACCEPTED AT A COLLEGE OF YOUR CHOICE, 



YOU EARNED A SCORE OF 68 ON THE VERBAL SECTION OF THE TEST, 
THE VERBAL SECTION OF THE TEST ^EASURES YOUR ABILITY TO READ 
WITH INEERSTANDING AND TO USE WORDS EFFECTIVELY, A SCORE AS HIGH 
OR HIGHER THAN THE ONE YOU HAVE EARNED IS EARNED BY LESS THAN II 
PER CENT OF JUNIORS OF YOUR SEX WHO LATER ENTER COLl£GE. \€RBAL 
APTITUDE IS PARTICULARLY IW>ORTANT FOR SUCCESSFUL COLLEGE WORK 
IN THE HUm«ITIES AND FINE ARTS, 

YOU EARNED A SCORE OF 68 ON THE fWHEMATICAL SECTION OF THE 
TEST, THE MATHEWTICAL SECTION OF THE TEST r€ASURES YOUR ABILITY 
TO REASON AND WORK EFFECTIVELY WITH NtTBERS. A SCORE AS HIGH OR 
HIGHER THAN THE ONE YOU HAVE EARNED IS EARNED BY LESS THAN II 
PER CENT OF JUNIORS OF YOUR SEX WHO LATER ENTER COLLEGE, 
MATHEMATICAL APTITUDE IS PARTICILARLY IM>ORTANT FOR SUCCESSFUL 
COLLEGE WORK IN THE SCIENCES AND ENGINEERING, 

YOU SHOULD NOT THINK OF YOUR TEST AS EXACT POINTS BUT AS A 
RANGE OF SCORES EXTENDING ABOUT THREE POINTS ABOVE AND THREE 
POINTS BELflW THE SCORE WE HAVE REPORTED TO YCU, TMEY GIVE A GOOD 
INDICATION OF HOW YOU WY EXPECT TO SCORE ON .'HE SCHOLASTIC 
APTITUBE TEST,. THE CHflCES ARE FOUR OUT OF FIVL ^HAT YOU WILL 
SCORE BETWEEN 530 AND 730 ON THE VERBAL SECTION OF THE 
SCHOLASTIC APTITUCE TEST WHEN YOU TAKE IT NEXT YEAR* THE CHANCES 
ARE FOUR our OF HVE THAT YOU WILL SCORE BETWEEN lo) AND 730 ON 
THE MATHEMATICAL SECTION OF THE SCHOLASTIC APTITLa*: TEST WHEN 
YOU TAKE IT NEXT YEAR, INSOFAR AS VERBAL AND MATHF.WICAL 
APTITUCES ARE CONCERNED YOU CAN HAVE CONFIDENCE IN YOUR ABIUTY 
TO DO SUCCESSFUL COLLEGE WORK, 

THE SCORES YOU HAVE EARNED SHOULD ENCOURAGE YOU TO APPLY 
FDR ADMISSION TO AN OUTSTANDING COLLEGE, YOUR FUTURE EDUCATIONAL 
PLANS SHOOD CONSIDER ADVANCED GRADIWTE WORK, AFTER YOU HAVE 
DISCUSSED YOUR SCORES WITH YOUR PARENTS AND YOUR COUNSELOR OR 
PRINCIPAL SHOULD YOU HAVE ADDITIONAL QUESTIONS ABOUT THEM YOU 
MAY WRITE TO EDUCATIONAL TESTING SERVICE/ PRINCETON, N. J, 

SINCERELY YOURS/ 
E, T, S, 
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Figure 3 

SAMPLE copy OF FINNEY'S REPORT ON THE MINNESOTA 
MULTI-PHASIC PERSONALITY INVENTORY 

This is a report of MMPI and CPl testing of a female age 16, case number B 000005. This test, like 
any test. Is subject to erro^ . Testing only supplements other diagnostic examinations. 

First let us examine the evidence of validity and the attitude with which she took the test. 

On the CPl she gives mostly the common and conventional answers. That may be 
a sign of at least average common sense and judgment, and of being sufficiently steady, 
reliable, and realistic. She does not give a consistently favorable nor a consistently 
unfavoraUe picture of herself. She is afrakJ to admit even small flaws in herself, in 
terms of standards which are naive, rigid, perfectionistic, moralistic unrealistic, and 
overly oonventk>nal. That shows a lack of insight. It also shows th . « ; distinguish 

clearly between fundamental obligations, which people can am 'pecc her to meet, 

and the lesser or shal lower ntiatters in which a falling short in performance is tolerable. 
She wants to make a good impression in taking the test, and she gives the Irr^pression of 
having at least an average degree of warmth. She is moderately ambitious, alert, and 
productive, and likes working. She has no serious doubts about herself. 

She does not tell of anxfety or stress and is not looking for help« She is a reasonably 
compliant person. She has a normal amount of flexibility. 

In terms of these factors, she seems to be a normal, average, flexible person. 
But the two-point code tells us as follows. She has hysterical conver$k>n reactions 
of some specific location or other. She is naive, exhibitk)nistic, self-centerad, and 
demanding, and tends to manipulate and exploit people. Because of repression she lacks 
insight and is not motivated for psychotherapy. 

Now, what is the evkience for psychosis or mental illness? 

None of the measures indicate that she is psychotic. Some measures are doubtful, 
a: follows. The obsessive and schizophrenic indicators are about at equal level. On a 
schizophrenic correction scale, she seems to have psychotic trends. But most measures 
indicate Mmx she is not psychotic. She does not use rituals or compulsive acts at all to 
ward off anxiety. 

Next we consider narcissism, guitt, and basic trust. 

Her self-esteem is low and she doesn't feel proud of herself. But by another measure 
her self-acceptance is within the average range, though she tends to blame herself a 
little more than the average. She shows signs of less than average guilt feeling. And 
she tends to deny guilt. She has a normal amount of concern with what people think 
of her. But she denies any feeling of self-consciousness or embarrassment* She has at 
least average dominance and initiative. She scores almost average on ego strength, 
and has fair tolerance for frustration. This is a good level of ego strength for a psychiatric 
patient. She has the assets of benefit from psychotherapy, but only if mot?vatk>n and 
distress are also present. To a moderate degree she maintains an optimistic atthude by 
denying discouragement. She tells of very little worrying; less than the average person. 
She shows signs of having some fears or phobias. But she does not admit fears or phobias. 
Now we turn our attention to problems of dependency. 

The signs are that she has only slightly more dependency need than the average, 
if at all so. Within the average or normal range she seems to put her dependency needs 
into action. 
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Now, what about baing demanding or orally aggressive? 
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